Thermoelectric-enhanced air and liquid cooling of an electronic system

ABSTRACT

Thermoelectric-enhanced air and liquid cooling of an electronic system is facilitated by providing a cooling apparatus which includes a liquid-cooled structure in thermal communication with an electronic component(s), and liquid-to-liquid and air-to-liquid heat exchangers coupled in series fluid communication via a coolant loop, which includes first and second loop portions coupled in parallel. The liquid-cooled structure is supplied coolant via the first loop portion, and a thermoelectric array is disposed with the first and second loop portions in thermal contact with first and second sides of the array. The thermoelectric array operates to transfer heat from coolant passing through the first loop portion to coolant passing through the second loop portion, and cools coolant passing through the first loop portion before the coolant passes through the liquid-cooled structure. Coolant passing through the first and second loop portions passes through the series-coupled heat exchangers, one of which functions as heat sink.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. Ser. No. 13/613,832, filedSep. 13, 2012, and entitled “Thermoelectric-Enhanced Air and LiquidCooling of an Electronic System”, and which is hereby incorporatedherein by reference in its entirety.

BACKGROUND

As is known, operating electronic components, such as electronicdevices, produce heat. This heat should be removed from the devices inorder to maintain device junction temperatures within desirable limits,with failure to remove heat effectively resulting in increased devicetemperatures, and potentially leading to thermal runaway conditions.Several trends in the electronics industry have combined to increase theimportance of thermal management, including heat removal for electronicdevices, including technologies where thermal management hastraditionally been less of a concern, such as CMOS. In particular, theneed for faster and more densely packed circuits has had a direct impacton the importance of thermal management. First, power dissipation, andtherefore heat production, increases as device operating frequenciesincrease. Second, increased operating frequencies may be possible atlower device junction temperatures. Further, as more and more devicesare packed onto a single chip, heat flux (Watts/cm²) increases,resulting in the need to remove more power from a given size chip ormodule. These trends have combined to create applications where it is nolonger desirable to remove heat from modern devices, and electronicsystems containing such devices, solely by traditional air coolingmethods, such as by using air cooled heat sinks with heat pipes or vaporchambers. Such air cooling techniques are inherently limited in theirability to extract heat from electronic components with moderate to highpower density.

BRIEF SUMMARY

The shortcomings of the prior art are overcome and additional advantagesare provided by, in one aspect, a method of fabricating a coolingapparatus is provided, which includes: providing a liquid-cooledstructure, the liquid-cooled structure being configured to couple to atleast one electronic component to be cooled; providing a coolant loop,the coolant loop comprising a first loop portion and a second loopportion, the first loop portion and the second loop portion beingparallel portions of the coolant loop; coupling the liquid-cooledstructure in fluid communication with the first loop portion of thecoolant loop; providing a liquid-to-liquid heat exchanger and anair-to-liquid heat exchanger coupled in series fluid communication viathe coolant loop, wherein coolant egressing from the liquid-to-liquidheat exchanger passes, via the coolant loop, through the air-to-liquidheat exchanger; and providing a thermoelectric array comprising at leastone thermoelectric module, the thermoelectric array being disposed withthe first loop portion of the coolant loop at least partially in thermalcontact with a first side of the thermoelectric array, and the secondloop portion of the coolant loop at least partially in thermal contactwith a second side of the thermoelectric array, wherein thethermoelectric array operates to transfer heat from coolant passingthrough the first loop portion to coolant passing through the secondloop portion, the thermoelectric array cooling coolant passing throughthe first loop portion before the coolant passes through theliquid-cooled structure, and after passing through the liquid-cooledstructure, the coolant passing through the first loop portion and thecoolant passing through the second loop portion passes through theseries-coupled, liquid-to-liquid heat exchanger and air-to-liquid heatexchanger, wherein one of the liquid-to-liquid heat exchanger or theair-to-liquid heat exchanger operates as heat sink for the coolant loop,dependent on a mode of operation of the cooling apparatus.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled computer installation;

FIG. 2 depicts a problem with conventional air-cooling of an electronicsrack, wherein recirculation airflow patterns may be established withinan air-cooled computer installation;

FIG. 3 is a cross-sectional plan view of one embodiment of anelectronics rack utilizing at least one air-to-liquid heat exchangerdisposed at the air outlet side of the electronics rack, in accordancewith one or more aspects of the present invention;

FIG. 4 is a front elevational view of one embodiment of a liquid-cooledelectronics rack comprising multiple electronic subsystems cooled by acooling apparatus, in accordance with one or more aspects of the presentinvention;

FIG. 5 is a schematic of one embodiment of an electronic subsystem of anelectronics rack, wherein an electronic module is liquid-cooled bysystem coolant provided by one or more modular cooling units associatedwith the electronics rack, in accordance with one or more aspects of thepresent invention;

FIG. 6 is a schematic of one embodiment of a modular cooling unit of aliquid-cooled electronics rack, in accordance with one or more aspectsof the present invention;

FIG. 7 is a plan view of one embodiment of an electronic subsystemlayout illustrating an air and liquid cooling subsystem for coolingcomponents of the electronic subsystem, in accordance with one or moreaspects of the present invention;

FIG. 8 depicts one detailed embodiment of a partially-assembledelectronic subsystem layout, wherein the electronic subsystem includeseight heat-generating electronic components to be actively cooled, eachhaving a respective liquid-cooled cold plate of a liquid-based coolingsystem coupled thereto, in accordance with one or more aspects of thepresent invention;

FIG. 9 is a schematic of one embodiment of a cooled electronic systemcomprising a liquid-cooled electronics rack and a cooling apparatusassociated therewith, wherein the cooling apparatus includes two modularcooling units (MCUs) for providing in parallel liquid coolant to theelectronic subsystems of the rack, and an air-to-liquid heat exchangerdisposed, for example, at an air outlet side of the electronics rack forcooling air egressing therefrom, in accordance with one or more aspectsof the present invention;

FIG. 10 is a schematic of one embodiment of heat transfer from one ormore MCUs disposed within one or more electronics racks of a data centerto a cooling tower disposed outside of the data center, in accordancewith one or more aspects of the present invention;

FIG. 11A is a schematic of an alternate embodiment of a cooledelectronic system comprising an electronics rack and a cooling apparatusassociated therewith shown in normal-mode, wherein system coolant flowsin parallel through the electronic subsystems and the air-to-liquid heatexchanger, in accordance with one or more aspects of the presentinvention;

FIG. 11B is a schematic of the cooled electronic system of FIG. 11A,shown in failure-mode, wherein the multiple isolation valves aretransitioned to establish serial flow of system coolant from theelectronic subsystems to the air-to-liquid heat exchanger, in accordancewith one or more aspects of the present invention;

FIG. 12 is a schematic of one embodiment of a control arrangement forcontrolling two MCUs of a cooling apparatus such as depicted in FIGS.11A & 11B, in accordance with one or more aspects of the presentinvention;

FIG. 13 depicts one embodiment of a process for controllingtransitioning of the cooling apparatus of FIGS. 11A & 11B betweennormal-mode, parallel flow of system coolant through the electronicsubsystems and the air-to-liquid heat exchanger, and failure-mode,serial flow of system coolant from the electronic subsystems to theair-to-liquid heat exchanger, in accordance with one or more aspects ofthe present invention;

FIG. 14 is a schematic of another embodiment of a cooled electronicsystem comprising, by way of example, an electronics rack and a coolingapparatus associated therewith, operable in either an air-cooled mode ora liquid-cooled mode, in accordance with one or more aspects of thepresent invention;

FIG. 15 is a cross-sectional elevational view of one embodiment of athermoelectric-enhanced, fluid-to-fluid heat exchange assembly for acooling apparatus such as depicted in FIG. 14, in accordance with one ormore aspects of the present invention;

FIG. 16 is a flowchart of one embodiment of a process for controllingoperation of the cooling apparatus depicted in FIG. 14, when in eitherair-cooled mode or liquid-cooled mode thereof, in accordance with one ormore aspects of the present invention; and

FIG. 17 depicts one embodiment of a computer program productincorporating one or more aspects of the present invention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, “rack-mounted electronicequipment”, and “rack unit” are used interchangeably, and unlessotherwise specified include any housing, frame, rack, compartment, bladeserver system, etc., having one or more heat generating components of acomputer system or electronic system, and may be, for example, astand-alone computer processor having high, mid or low end processingcapability. In one embodiment, an electronics rack may comprise multipleelectronic systems or subsystems, each having one or more heatgenerating components disposed therein requiring cooling. “Electronicsystem” or “electronic subsystem” refers to any sub-housing, assembly,board, blade, book, drawer, node, etc., having one or more heatgenerating electronic components disposed therein or thereon. Eachelectronic system or subsystem of an electronics rack may be movable orfixed relative to the electronics rack, with the rack-mounted electronicdrawers of a multi-drawer rack unit and blades of a blade center systembeing two examples of systems or subsystems of an electronics rack to becooled.

“Electronic component” refers to any heat generating electroniccomponent of, for example, a computer system or other electronics unitrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies and/or other electronicdevices to be cooled, including one or more processor dies, memory diesand memory support dies. As a further example, the electronic componentmay comprise one or more bare dies or one or more packaged dies disposedon a common carrier. As used herein, “primary heat generating component”refers to a primary heat generating electronic component within anelectronic subsystem, while “secondary heat generating component” refersto an electronic component of the electronic subsystem generating lessheat than the primary heat generating component to be cooled. “Primaryheat generating die” refers, for example, to a primary heat generatingdie or chip within a heat generating electronic component comprisingprimary and secondary heat generating dies (with a processor die beingone example). “Secondary heat generating die” refers to a die of amulti-die electronic component generating less heat than the primaryheat generating die thereof (with memory dies and memory support diesbeing examples of secondary dies to be cooled). As one example, a heatgenerating electronic component could comprise multiple primary heatgenerating bare dies and multiple secondary heat generating dies on acommon carrier. Further, unless otherwise specified herein, the terms“liquid-cooled cold plate” and “liquid-cooled structure” refer to anythermally conductive structure having a plurality of channels orpassageways formed therein for flowing of liquid coolant therethrough.In addition, “metallurgically bonded” refers generally herein to twocomponents being welded, brazed or soldered together by any means.

As used herein, “air-to-liquid heat exchanger” means any heat exchangemechanism characterized as described herein through which liquid coolantcan circulate; and includes, one or more discrete air-to-liquid heatexchangers coupled either in series or in parallel. An air-to-liquidheat exchanger may comprise, for example, one or more coolant flowpaths, formed of thermally conductive tubing (such as copper or othertubing) in thermal or mechanical contact with (for example) a pluralityof air-cooled cooling fins. Size, configuration and construction of theair-to-liquid heat exchanger can vary without departing from the scopeof the invention disclosed herein. A “liquid-to-liquid heat exchanger”may comprise, for example, two or more coolant flow paths, formed ofthermally conductive tubing (such as copper or other tubing) in thermalor mechanical contact with each other. Size, configuration andconstruction of the liquid-to-liquid heat exchanger can vary withoutdeparting from the scope of the invention disclosed herein. Further,“data center” refers to a computer installation containing one or moreelectronics racks to be cooled. As a specific example, a data center mayinclude one or more rows of rack-mounted computing units, such as serverunits.

One example of facility coolant and system coolant discussed herein iswater. However, the concepts disclosed herein are readily adapted to usewith other types of coolant on the facility side and/or on the systemside. For example, one or more of the coolants may comprise a brine, afluorocarbon liquid, a liquid metal, or other similar coolant, or arefrigerant, while still maintaining the advantages and unique featuresof the present invention.

Reference is made below to the drawings, which are not drawn to scalefor ease of understanding of the various aspects of the presentinvention, wherein the same reference numbers used throughout differentfigures designate the same or similar components.

As shown in FIG. 1, in a raised floor layout of an air cooled computerinstallation 100 typical in the prior art, multiple electronics racks110 are disposed in one or more rows. A computer installation such asdepicted in FIG. 1 may house several hundred, or even several thousandmicroprocessors. In the arrangement of FIG. 1, chilled air enters thecomputer room via floor vents from a supply air plenum 145 definedbetween the raised floor 140 and a base or sub-floor 165 of the room.Cooled air is taken in through louvered covers at air inlet sides 120 ofthe electronics racks and expelled through the backs, i.e., air outletsides 130, of the electronics racks. Each electronics rack 110 may havean air moving device (e.g., fan or blower) to provide forcedinlet-to-outlet air flow to cool the electronic components within thedrawer(s) of the rack. The supply air plenum 145 provides conditionedand cooled air to the air-inlet sides of the electronics racks viaperforated floor tiles 160 disposed in a “cold” aisle of the computerinstallation. The conditioned and cooled air is supplied to plenum 145by one or more conditioned air units 150, also disposed within thecomputer installation 100. Room air is taken into each conditioned airunit 150 near an upper portion thereof. This room air comprises in partexhausted air from the “hot” aisles of the computer installation definedby opposing air outlet sides 130 of the electronics racks 110.

Due to the ever increasing air flow requirements through electronicsracks, and limits of air distribution within the typical computer roominstallation, recirculation problems within the room may occur. This isshown in FIG. 2 for a conventional raised floor layout, wherein hot airrecirculation 200 occurs from the air outlet sides 130 of theelectronics racks back to the cold air aisle defined by the opposing airinlet sides 120 of the electronics racks. This recirculation can occurbecause the conditioned air supplied through tiles 160 is typically onlya fraction of the air flow rate forced through the electronics racks bythe air moving devices disposed therein. This can be due, for example,to limitations on the tile sizes (or diffuser flow rates). The remainingfraction of the supply of inlet side air is often made up by ambientroom air through recirculation 200. This re-circulating flow is oftenvery complex in nature, and can lead to significantly higher rack unitinlet temperatures than might be expected.

The recirculation of hot exhaust air from the hot aisle of the computerroom installation to the cold aisle can be detrimental to theperformance and reliability of the computer system(s) or electronicsystem(s) within the racks. Data center equipment is typically designedto operate with rack air inlet temperatures in the 18-35° C. range. Fora raised floor layout such as depicted in FIG. 1, however, temperaturescan range from 15-20° C. at the lower portion of the rack, close to thecooled air input floor vents, to as much as 45-50° C. at the upperportion of the electronics rack, where the hot air can form aself-sustaining recirculation loop. Since the allowable rack heat loadis limited by the rack inlet air temperature at the “hot” part, thistemperature distribution correlates to a lower processing capacity.Also, computer installation equipment almost always represents a highcapital investment to the customer. Thus, it is of significantimportance, from a product reliability and performance view point, andfrom a customer satisfaction and business perspective, to maintain thetemperature of the inlet air uniform. The efficient cooling of suchcomputer and electronic systems, and the amelioration of localized hotair inlet temperatures to one or more rack units due to recirculation ofair currents, are addressed by the apparatuses and methods disclosedherein.

FIG. 3 depicts one embodiment of a cooled electronic system, generallydenoted 300, in accordance with an aspect of the present invention. Inthis embodiment, electronic system 300 includes an electronics rack 310having an inlet door cover 320 and an outlet door cover 330 which haveopenings to allow for the ingress and egress of external air from theinlet side to the outlet side of the electronics rack 310. The systemfurther includes at least one air moving device 312 for moving externalair across at least one electronic drawer unit 314 positioned within theelectronics rack. Disposed within outlet door cover 330 is a heatexchange assembly 340. Heat exchange assembly 340 includes anair-to-liquid heat exchanger through which the inlet-to-outlet air flowthrough the electronics rack passes. A computer room water conditioner(CRWC) 350 is used to buffer heat exchange assembly 340 from thebuilding utility or local chiller coolant 360, which is provided asinput to CRWC 350. The CRWC 350 provides system water or system coolantto heat exchange assembly 340. Heat exchange assembly 340 removes heatfrom the exhausted inlet-to-outlet air flow through the electronics rackfor transfer via the system water or coolant to CRWC 350.Advantageously, providing a heat exchange assembly with an air-to-liquidheat exchanger such as disclosed herein at the outlet door cover of oneor more electronics racks in a computer installation can, in normaloperation, significantly reduce heat loads on existing air conditioningunits within the computer installation, and facilitate the cooling ofthe rack-mounted electronics units.

FIG. 4 depicts one embodiment of a liquid-cooled electronics rack 400which employs a cooling apparatus to be monitored and operated asdescribed herein. In one embodiment, liquid-cooled electronics rack 400comprises a plurality of electronic subsystems 410, which are (in oneembodiment) processor or server nodes. A bulk power regulator 420 isshown disposed at an upper portion of liquid-cooled electronics rack400, and two modular cooling units (MCUs) 430 are disposed in a lowerportion of the liquid-cooled electronics rack. In the embodimentsdescribed herein, the coolant is assumed to be water or an aqueous-basedsolution, again, by way of example only.

In addition to MCUs 430, the cooling apparatus includes a system watersupply manifold 431, a system water return manifold 432, andmanifold-to-node fluid connect hoses 433 coupling system water supplymanifold 431 to electronic subsystems 410, and node-to-manifold fluidconnect hoses 434 coupling the individual electronic subsystems 410 tosystem water return manifold 432. Each MCU 430 is in fluid communicationwith system water supply manifold 431 via a respective system watersupply hose 435, and each MCU 430 is in fluid communication with systemwater return manifold 432 via a respective system water return hose 436.

As illustrated, heat load of the electronic subsystems is transferredfrom the system water to cooler facility water supplied by facilitywater supply line 440 and facility water return line 441 disposed, inthe illustrated embodiment, in the space between a raised floor 145 anda base floor 165.

FIG. 5 schematically illustrates operation of the cooling apparatus ofFIG. 4, wherein a liquid-cooled cold plate 500 is shown coupled to anelectronic module 501 of an electronic subsystem 410 within theliquid-cooled electronics rack 400. Heat is removed from electronicmodule 501 via the system coolant circulated via pump 520 through coldplate 500 within the system coolant loop defined by liquid-to-liquidheat exchanger 521 of modular cooling unit 430, lines 522, 523 and coldplate 500. The system coolant loop and modular cooling unit are designedto provide coolant of a controlled temperature and pressure, as well ascontrolled chemistry and cleanliness to cool the electronic module(s).Furthermore, the system coolant is physically separate from the lesscontrolled facility coolant in lines 440, 441, to which heat isultimately transferred.

FIG. 6 depicts a more detailed embodiment of a modular cooling unit 430,in accordance with an aspect of the present invention. As shown in FIG.6, modular cooling unit 430 includes a first coolant loop whereinbuilding chilled, facility coolant is supplied 610 and passes through acontrol valve 620 driven by a motor 625. Valve 620 determines an amountof facility coolant to be passed through heat exchanger 521, with aportion of the facility coolant possibly being returned directly via abypass orifice 635. The modular water cooling unit further includes asecond coolant loop with a reservoir tank 640 from which system coolantis pumped, either by pump 650 or pump 651, into the heat exchanger 521for conditioning and output thereof, as cooled system coolant to theelectronics rack to be cooled. The cooled system coolant is supplied tothe system water supply manifold and system water return manifold of theliquid-cooled electronics rack via the system water supply hose 435 andsystem water return hose 436, respectively.

FIG. 7 depicts one embodiment of an electronic subsystem 410 componentlayout wherein one or more air moving devices 711 provide forced airflow 715 to cool multiple components 712 within electronic subsystem410. Cool air is taken in through a front 731 and exhausted out a back733 of the drawer. The multiple components to be cooled include multipleprocessor modules to which liquid-cooled cold plates 720 (of aliquid-based cooling system) are coupled, as well as multiple arrays ofmemory modules 730 (e.g., dual in-line memory modules (DIMMs)) andmultiple rows of memory support modules 732 (e.g., DIMM control modules)to which air-cooled heat sinks are coupled. In the embodimentillustrated, memory modules 730 and the memory support modules 732 arepartially arrayed near front 731 of electronic subsystem 410, andpartially arrayed near back 733 of electronic subsystem 410. Also, inthe embodiment of FIG. 7, memory modules 730 and the memory supportmodules 732 are cooled by air flow 715 across the electronic subsystem.

The illustrated liquid-based cooling system further includes multiplecoolant-carrying tubes connected to and in fluid communication withliquid-cooled cold plates 720. The coolant-carrying tubes comprise setsof coolant-carrying tubes, with each set including (for example) acoolant supply tube 740, a bridge tube 741 and a coolant return tube742. In this example, each set of tubes provides liquid coolant to aseries-connected pair of cold plates 720 (coupled to a pair of processormodules). Coolant flows into a first cold plate of each pair via thecoolant supply tube 740 and from the first cold plate to a second coldplate of the pair via bridge tube or line 741, which may or may not bethermally conductive. From the second cold plate of the pair, coolant isreturned through the respective coolant return tube 742.

FIG. 8 depicts in greater detail an alternate electronics drawer layoutcomprising eight processor modules, each having a respectiveliquid-cooled cold plate of a liquid-based cooling system coupledthereto. The liquid-based cooling system is shown to further includeassociated coolant-carrying tubes for facilitating passage of liquidcoolant through the liquid-cooled cold plates and a header subassemblyto facilitate distribution of liquid coolant to and return of liquidcoolant from the liquid-cooled cold plates. By way of specific example,the liquid coolant passing through the liquid-based cooling subsystem ischilled water.

As noted, various liquid coolants significantly outperform air in thetask of removing heat from heat generating electronic components of anelectronic system, and thereby more effectively maintain the componentsat a desirable temperature for enhanced reliability and peakperformance. As liquid-based cooling systems are designed and deployed,it is advantageous to architect systems which maximize reliability andminimize the potential for leaks while meeting all other mechanical,electrical and chemical requirements of a given electronics systemimplementation. These more robust cooling systems have unique problemsin their assembly and implementation. For example, one assembly solutionis to utilize multiple fittings within the electronic system, and useflexible plastic or rubber tubing to connect headers, cold plates, pumpsand other components. However, such a solution may not meet a givencustomer's specifications and need for reliability.

Thus, presented herein in one aspect (and by way of example only) is arobust and reliable liquid-based cooling system specially preconfiguredand prefabricated as a monolithic structure for positioning within aparticular electronics drawer.

FIG. 8 is an isometric view of one embodiment of an electronic drawerand monolithic cooling system, in accordance with an aspect of thepresent invention. The depicted planar server assembly includes amulti-layer printed circuit board to which memory DIMM sockets andvarious electronic components to be cooled are attached both physicallyand electrically. In the cooling system depicted, a supply header isprovided to distribute liquid coolant from a single inlet to multipleparallel coolant flow paths and a return header collects exhaustedcoolant from the multiple parallel coolant flow paths into a singleoutlet. Each parallel coolant flow path includes one or more cold platesin series flow arrangement to cool one or more electronic components towhich the cold plates are mechanically and thermally coupled. The numberof parallel paths and the number of series-connected liquid-cooled coldplates depends, for example on the desired device temperature, availablecoolant temperature and coolant flow rate, and the total heat load beingdissipated from each electronic component.

More particularly, FIG. 8 depicts a partially assembled electronicsystem 813 and an assembled liquid-based cooling system 815 coupled toprimary heat generating components (e.g., including processor dies) tobe cooled. In this embodiment, the electronic system is configured for(or as) an electronic drawer of an electronics rack, and includes, byway of example, a support substrate or planar board 805, a plurality ofmemory module sockets 810 (with the memory modules (e.g., dual in-linememory modules) not shown), multiple rows of memory support modules 832(each having coupled thereto an air-cooled heat sink 834), and multipleprocessor modules (not shown) disposed below the liquid-cooled coldplates 820 of the liquid-based cooling system 815.

In addition to liquid-cooled cold plates 820, liquid-based coolingsystem 815 includes multiple coolant-carrying tubes, including coolantsupply tubes 840 and coolant return tubes 842 in fluid communicationwith respective liquid-cooled cold plates 820. The coolant-carryingtubes 840, 842 are also connected to a header (or manifold) subassembly850 which facilitates distribution of liquid coolant to the coolantsupply tubes and return of liquid coolant from the coolant return tubes842. In this embodiment, the air-cooled heat sinks 834 coupled to memorysupport modules 832 closer to front 831 of electronic drawer 813 areshorter in height than the air-cooled heat sinks 834′ coupled to memorysupport modules 832 near back 833 of electronics drawer 813. This sizedifference is to accommodate the coolant-carrying tubes 840, 842 since,in this embodiment, the header subassembly 850 is at the front 831 ofthe electronic drawer and the multiple liquid-cooled cold plates 820 arein the middle of the drawer.

Liquid-based cooling system 815 comprises (in this embodiment) apreconfigured monolithic structure which includes multiple(pre-assembled) liquid-cooled cold plates 820 configured and disposed inspaced relation to engage respective heat generating electroniccomponents. Each liquid-cooled cold plate 820 includes, in thisembodiment, a liquid coolant inlet and a liquid coolant outlet, as wellas an attachment subassembly (i.e., a cold plate/load arm assembly).Each attachment subassembly is employed to couple its respectiveliquid-cooled cold plate 820 to the associated electronic component toform the cold plate and electronic component assemblies. Alignmentopenings (i.e., thru-holes) are provided on the sides of the cold plateto receive alignment pins or positioning dowels during the assemblyprocess. Additionally, connectors (or guide pins) are included withinattachment subassembly which facilitate use of the attachment assembly.

As shown in FIG. 8, header subassembly 850 includes two liquidmanifolds, i.e., a coolant supply header 852 and a coolant return header854, which in one embodiment, are coupled together via supportingbrackets. In the monolithic cooling structure of FIG. 8, the coolantsupply header 852 is metallurgically bonded and in fluid communicationto each coolant supply tube 840, while the coolant return header 854 ismetallurgically bonded and in fluid communication to each coolant returntube 852. A single coolant inlet 851 and a single coolant outlet 853extend from the header subassembly for coupling to the electronicsrack's coolant supply and return manifolds (not shown).

FIG. 8 also depicts one embodiment of the preconfigured,coolant-carrying tubes. In addition to coolant supply tubes 840 andcoolant return tubes 842, bridge tubes or lines 841 are provided forcoupling, for example, a liquid coolant outlet of one liquid-cooled coldplate to the liquid coolant inlet of another liquid-cooled cold plate toconnect in series fluid flow the cold plates, with the pair of coldplates receiving and returning liquid coolant via a respective set ofcoolant supply and return tubes. In one embodiment, the coolant supplytubes 840, bridge tubes 841 and coolant return tubes 842 are eachpreconfigured, semi-rigid tubes formed of a thermally conductivematerial, such as copper or aluminum, and the tubes are respectivelybrazed, soldered or welded in a fluid-tight manner to the headersubassembly and/or the liquid-cooled cold plates. The tubes arepreconfigured for a particular electronic system to facilitateinstallation of the monolithic structure in engaging relation with theelectronic system.

Liquid cooling of heat-generating electronic components within anelectronics rack can greatly facilitate removal of heat generated bythose components. However, in certain high performance systems, the heatdissipated by certain components being liquid-cooled, such asprocessors, may exceed the ability of the liquid cooling system toextract heat. For example, a fully configured liquid-cooled electronicsrack, such as described hereinabove may dissipate approximately 72 kW ofheat. Half of this heat may be removed by liquid coolant usingliquid-cooled cold plates such as described above. The other half of theheat may be dissipated by memory, power supplies, etc., which areair-cooled. Given the density at which electronics racks are placed on adata center floor, existing air-conditioning facilities are stressedwith such a high air heat load from the electronics rack. Thus, asolution presented herein is to incorporate an air-to-liquid heatexchanger, for example, at the air outlet side of the electronics rack,to extract heat from air egressing from the electronics rack. Thissolution is presented herein in combination with liquid-cooled coldplate cooling of certain primary heat-generating components within theelectronics rack. To provide the necessary amount of coolant, two MCUs(in one embodiment) may be associated with the electronics rack, andsystem coolant would be fed from each MCU to the air-to-liquid heatexchanger in parallel to the flow of system coolant to the liquid-cooledcold plates disposed within the one or more electronic subsystems of theelectronics rack.

Also, for a high availability system, techniques are describedhereinbelow for maintaining operation of one modular cooling unit,notwithstanding failure of another modular cooling unit of anelectronics rack. This allows continued provision of system coolant tothe one or more electronic subsystems of the rack being liquid-cooled.To facilitate liquid cooling of the primary heat-generating electronicscomponents within the electronics rack, one or more isolation valves areemployed, in one embodiment upon detection of failure at one MCU of thetwo MCUs, to shut off coolant flow to the air-to-liquid heat exchanger,and thereby, conserve coolant for the direct cooling of the electronicsubsystems.

The above-summarized aspects of the invention are described furtherbelow with reference to the embodiment of FIG. 9.

FIG. 9 illustrates one embodiment of a system wherein an electronicsrack 900 includes a plurality of heat-generating electronic subsystems910, which are liquid-cooled employing a cooling apparatus comprising atleast two modular cooling units (MCUs) 920, 930 labeled MCU 1 & MCU 2,respectively. The MCUs are configured and coupled to provide systemcoolant in parallel to the plurality of heat-generating electronicsubsystems for facilitating liquid cooling thereof. Each MCU 920, 930includes a liquid-to-liquid heat exchanger 921, 931, coupled to a firstcoolant loop 922, 932, and to a second coolant loop, 923, 933,respectively. The first coolant loops 922, 932 are coupled to receivechilled coolant, such as facility coolant, via (for example) facilitywater supply line 440 and facility water return line 441. Each firstcoolant loop 922, 932 passes at least a portion of the chilled coolantflowing therein through the respective liquid-to-liquid heat exchanger921, 931. Each second coolant loop 923, 933 provides cooled systemcoolant to the plurality of heat-generating electronic subsystems 910 ofelectronics rack 900, and expels heat via the respectiveliquid-to-liquid heat exchanger 921, 931 from the plurality ofheat-generating electronic subsystems 910 to the chilled coolant in thefirst coolant loop 922, 932.

The second coolant loops 923, 933 include respective coolant supplylines 924, 934, which supply cooled system coolant from theliquid-to-liquid heat exchangers 921, 931 to a system coolant supplymanifold 940. System coolant supply manifold 940 is coupled via flexiblesupply hoses 941 to the plurality of heat-generating electronicsubsystems 910 of electronics rack 900 (e.g., using quick connectcouplings connected to respective ports of the system coolant supplymanifold). Similarly, second coolant loops 923, 933 include systemcoolant return lines 925, 935 coupling a system coolant return manifold950 to the respective liquid-to-liquid heat exchangers 921, 931. Systemcoolant is exhausted from the plurality of heat-generating electronicsubsystems 910 via flexible return hoses 951 coupling theheat-generating electronic subsystems to system coolant return manifold950. In one embodiment, the return hoses may couple to respective portsof the system coolant return manifold via quick connect couplings.Further, in one embodiment, the plurality of heat-generating electronicsubsystems each include a respective liquid-based cooling subsystem,such as described above in connection with FIGS. 7 & 8, coupled toflexible supply hoses 941 and flexible return hoses 951 to facilitateliquid cooling of one or more heat-generating electronic componentsdisposed within the electronic subsystem.

In addition to supplying and exhausting system coolant in parallel tothe plurality of heat-generating electronic subsystems of theelectronics rack, the MCUs 920, 930 also provide in parallel systemcoolant to an air-to-liquid heat exchanger 960 disposed, for example,for cooling air passing through the electronics rack from an air inletside to an air outlet side thereof. By way of example, air-to-liquidheat exchanger 960 is a rear door heat exchanger disposed at the airoutlet side of electronics rack 900. Further, in one example,air-to-liquid heat exchanger 960 is sized to cool substantially all airegressing from electronics rack 900, and thereby reduce air-conditioningrequirements for a data center containing the electronics rack. In oneexample, a plurality of electronics racks in the data center are eachprovided with a cooling apparatus such as described herein and depictedin FIG. 9.

In the embodiment of FIG. 9, system coolant flows to and fromair-to-liquid heat exchanger 960 via a coolant supply line 961 couplingsystem coolant supply manifold 940 to air-to-liquid heat exchanger 960,and a coolant return line 962 coupling the air-to-liquid heat exchangerto system coolant return manifold 950. Quick connect couplings may beemployed at the inlet and outlet of air-to-liquid heat exchanger 960and/or at corresponding ports at the system coolant supply and returnmanifolds to facilitate connection of coolant supply and return lines961, 962. In one embodiment, it is assumed that one MCU of the two MCUsillustrated is incapable of being sized to function within requireddesign parameters as a primary MCU (with the other MCU being a backupMCU) to extract the full heat load from both the plurality ofheat-generating electronic subsystems and the air-to-liquid heatexchanger. Therefore, the two MCUs 920, 930 are assumed in normaloperation to be functioning in parallel. This also ensures a measure ofredundancy to the cooling system.

As shown, the cooling system further includes a system controller 970,and an MCU control 1 980 and an MCU control 2 990, which cooperatetogether to monitor system coolant temperature of each MCU, andautomatically isolate air-to-liquid heat exchanger 960 upon detection offailure of one MCU (as well as to ensure shut down of a failing MCU) soas not to degrade cooling capability of the system coolant provided bythe remaining operational MCU to the electronics subsystems of the rack.In one embodiment, the MCU control 1 and the MCU control 2 are controlcards, each associated with a respective MCU.

As shown, system controller 970 is coupled to both MCU control 1 and theMCU control 2. MCU control 1 980 is coupled to a temperature sensor T1981, which is disposed to sense system coolant temperature within systemcoolant supply line 924, for example, near a coolant outlet ofliquid-to-liquid heat exchanger 921 within MCU 1 920. Additionally, MCUcontrol 1 980 is coupled to a solenoid-actuated isolation valve S1 982,which in the embodiment depicted, is disposed within coolant supply line961 coupling in fluid communication system coolant supply manifold 940to air-to-liquid heat exchanger 960. Similarly, MCU control 2 990couples to MCU 2 930, as well as to a second temperature sensor T2 991,disposed for sensing system coolant temperature within system coolantsupply line 934, and to a second isolation valve S2 992, which in theexample depicted, is coupled to coolant return line 962 couplingair-to-liquid heat exchanger 960 to system coolant return manifold 950.

Also note that in the example of FIG. 9, the MCUs operate to transferheat extracted by the circulating system coolant to the facility chilledcoolant. Note also that system coolant flow to the electronic subsystemsand the air-to-liquid heat exchanger is in parallel. This flowarrangement advantageously provides a lowest temperature coolant to allof the cooling components in the system. This in turn translates intolowest possible electronic component temperatures within the electronicsubsystems, as well as a maximum amount of heat removal from air flowingthrough the electronics rack by the air-to-liquid heat exchanger, forexample, to allow a substantial amount of the heat to be removed priorto returning the air to the computer room environment.

FIG. 10 is a high-level illustration of one embodiment of heat transferthrough a data center cooling system comprising liquid-cooledelectronics racks such as described herein. In this embodiment, heat istransferred from one or more electronics racks within a data center 1000to a facilities area 1010, and ultimately to an area 1020 outside of thefacilities area and the data center. Specifically, one or more coolingunits, such as modular cooling units (MCUs) 1001, each comprise aliquid-to-liquid heat exchanger for facilitating transfer of heat fromsystem coolant flowing through the associated liquid-cooled electronicsrack to a facility coolant loop 1011 disposed (in this embodiment) totransfer heat between MCU 1001 and a refrigeration chiller 1012. Acoolant pump 1013 pumps facility coolant through facility coolant loop1011 to facilitate transfer of heat from the liquid-to-liquid heatexchanger within MCU 1001 to an evaporator 1014 within refrigerationchiller 1012. Evaporator 1014 extracts heat from facility coolantflowing through facility coolant loop 1011 and transfers the heat to arefrigerant flowing through a refrigerant loop 1015. Refrigerant loop1015 couples in fluid communication evaporator 1014, a compressor 1016,a condenser 1017 and an expansion valve 1018. Refrigeration chiller 1012implements, in one embodiment, a conventional vapor-compressionrefrigeration cycle. Condenser 1017 dissipates heat to, for example, acondenser water loop 1021 disposed between refrigeration chiller 1012and a cooling tower 1022 positioned, for example, outside 1020 facilityarea 1010 and data center 1000. Vaporized water is condensed withincooling tower 1022 and the condensate is re-circulated via a condensatewater pump 1023 through condenser 1017 of refrigeration chiller 1012.

Thus, the overall cooling system transfers heat from the IT equipment,i.e., the electronics rack, to the outdoor ambient air. Moving in thedirection of heat flow, heat generated within the electronics rack istransferred to the facility coolant loop via the modular coolingunit(s). The facility coolant loop carries the heat to a refrigerationchiller, with the heat being taken into the refrigeration chiller at itsevaporator and rejected to a condenser water loop at its condenser. Thecondenser water passes outside of the facility to, for example, one ormore cooling towers that transfer the heat to the outside ambient air.There are numerous events that could occur which could lead to eitherthe loss of facility coolant cooling within facility coolant loop 1011,or the loss of facility coolant flow within facility coolant loop 1011.The former could result, for example, if the refrigeration chiller goesoff-line, but the facility coolant pump continues to pump facilitycoolant through facility coolant loop 1011, while the later might resultfrom losing operation of the facility coolant pump 1013. Either eventcould lead to an over-temperature condition within the one or moreliquid-cooled electronics racks within the data center being serviced bythe refrigeration chiller, possibly resulting in shutting down one ormore of the electronics racks.

Disclosed herein with reference to FIGS. 11A-13 is one embodiment of areconfigurable cooling apparatus which allows continued operation of theelectronic subsystems of an electronics rack, notwithstanding loss offacility coolant cooling or loss of facility coolant flow. In the eventof loss of facility coolant cooling or flow, heat extracted by systemcoolant flowing through the liquid-cooled electronic subsystems isrejected to the data center room air via the air-to-liquid heatexchanger. In this mode, temperature of the system coolant will riseuntil an equilibrium is reached where the amount of heat transferred tothe room air equals the amount of heat extracted from the electronicsubsystems. Should the air-to-liquid heat exchanger be coupled inparallel with the electronic subsystems, as depicted for example in FIG.9, the equilibrium temperature of the fluid at the electronic subsystemsbeing cooled would be higher than it would be if the air-to-liquid heatexchanger was coupled in series with the electronic subsystems(described below with reference to FIG. 11B). That is, in optimal,normal-mode operation, the air-to-liquid heat exchanger is coupled inparallel with the liquid-cooled electronic subsystems, while infailure-mode operation (e.g., due to a loss of facility coolant coolingor facility coolant flow), it is optimal for the air-to-liquid heatexchanger to be coupled in series fluid communication with the outletsof the liquid-cooled electronic subsystems. Therefore, disclosedhereinbelow (in one aspect) is a cooling apparatus, including a valveand piping arrangement, together with control processes, thatautomatically adjusts flow of system coolant through the coolingapparatus between normal-mode, parallel flow of system coolant throughthe electronic subsystems and the air-to-liquid heat exchanger, and afailure-mode, serial flow of system coolant from the electronicsubsystems to the air-to-liquid heat exchanger.

FIGS. 11A & 11B respectively depict normal-mode and failure-modeoperation of one embodiment of such a cooling apparatus. This coolingapparatus embodiment has certain changes to the piping and isolationvalves (as described below) compared with the embodiment of FIG. 9.

In the embodiment of FIGS. 11A & 11B, a cooled electronic system isprovided comprising an electronics rack 1100 including a plurality ofheat-generating electronic subsystems 1110, which are liquid-cooledemploying a cooling apparatus comprising, in the embodiment depicted,two modular cooling units (MCUs) 1120, 1130, labeled MCU 1 & MCU 2,respectively. Note that in this example, although described withreference to two MCUs, the cooling apparatus could be implemented usinga single MCU sized to function within the required normal-mode designparameters for cooling the plurality of heat-generating electronicsubsystems and the air-to-liquid heat exchanger. In the dual MCUembodiment depicted in FIGS. 11A & 11B, the two MCUs illustrated mayoperate as a primary MCU, and a backup MCU, or may operate in parallel(e.g., in the case where neither MCU can extract the full heat load fromthe plurality of heat-generating electronic subsystems and theair-to-liquid heat exchanger).

The MCUs are configured and coupled to provide system coolant inparallel to the plurality of heat-generating electronic subsystems forfacilitating liquid-cooling thereof. Each MCU 1120, 1130 includes aliquid-to-liquid heat exchanger 1128, 1138, coupled to a facilitycoolant loop 1122, 1132 and to a system coolant loop 1123, 1133,respectively. Each MCU further includes a reservoir tank 1126, 1136, asystem coolant pump 1127, 1137, and a check valve 1129, 1139,respectively. The facility coolant loops 1122, 1132 are coupled toreceive chilled coolant, such as facility coolant, via (for example) afacility coolant supply line and a facility coolant return line (notshown). Each facility coolant loop 1122, 1132 includes a proportionalvalve P1, P2 for passing at least a portion of the chilled facilitycoolant flowing therein through the respective liquid-to-liquid heatexchangers 1128, 1138.

Each system coolant loop 1123, 1133 provides cooled system coolant tothe plurality of heat-generating electronic subsystems 1110 ofelectronics rack 1100, and expels heat via the respectiveliquid-to-liquid heat exchanger 1128, 1138 from the plurality ofheat-generating electronic subsystems 1110 to the chilled facilitycoolant in the facility coolant loop 1122, 1132, respectively. Thesystem coolant loops 1123, 1133 include respective coolant supply lines1124, 1134, which supply cooled system coolant from the liquid-to-liquidheat exchangers 1128, 1138 to a system coolant supply manifold 1140.System coolant supply manifold 1140 is coupled via, for example,flexible supply hoses, to the plurality of heat-generating electronicsubsystems 1110 of electronics rack 1100 (e.g., using quick connectcouplings coupled to respective ports of the system coolant supplymanifold). Similarly, system coolant loops 1123, 1133 include systemcoolant return lines 1125, 1135 coupling a system coolant returnmanifold 1150 to the respective liquid-to-liquid heat exchangers 1128,1138. System coolant is exhausted from the plurality of heat-generatingelectronic subsystems 1110 via flexible return hoses coupling theheat-generating electronic subsystems to the system coolant returnmanifold 1150. In one embodiment, the return hoses may couple torespective ports of the system coolant return manifold via quick connectcouplings. Further, in one embodiment, the plurality of heat-generatingelectronic subsystems 1110 each include a respective liquid-basedcooling subsystem, such as described above in connection with FIGS. 7 &8 coupled to facilitate liquid-cooling of one or more heat-generatingelectronic components disposed within the electronic subsystem.

In addition to supplying the system coolant in parallel to the pluralityof heat-generating electronic subsystems of the electronics rack, theMCUs 1120, 1130 also provide in parallel thereto (i.e., in normal-modeoperation) coolant to an air-to-liquid heat exchanger 1160 disposed, forexample, for cooling air passing through the electronics rack from anair inlet side to an air outlet side thereof. By way of example,air-to-liquid heat exchanger 1160 is a rear door heat exchanger disposedat the air outlet side of electronics rack 1100. Further, in oneexample, air-to-liquid heat exchanger 1160 is sized to at leastpartially cool all air egressing from electronics rack 1100, and therebyreduce air-conditioning requirements for a data center containing theelectronics rack. In one example, a plurality of electronics racks inthe data center are each provided with a cooling apparatus, such asdescribed herein and depicted in FIGS. 11A & 11B.

One embodiment of a control arrangement for the cooling apparatus ofFIGS. 11A & 11B is depicted in FIG. 12. In this embodiment, the coolingapparatus further includes a system controller 1200, and an MCU control1 1210 and an MCU control 2 1220, which (in one embodiment) cooperatetogether to monitor coolant temperature of each MCU (e.g., systemcoolant temperature), as well as flow of facility coolant through eachMCU, and to automatically transition the cooling apparatus betweennormal-mode, parallel flow of system coolant through the one or moreelectronic subsystems and the air-to-liquid heat exchanger tofailure-mode, serial flow of system coolant from the one or moreelectronic subsystems to the air-to-liquid heat exchanger, for example,responsive to direct or indirect detection of failure of the chilledfacility coolant within the facility coolant loop. Transitioning tofailure-mode responsive to detection of a facility coolant failureadvantageously establishes serial flow of system coolant from the atleast one electronic subsystem to the air-to-liquid heat exchanger, andtherefore, the rejecting of heat via the system coolant, from the atleast one electronic subsystem to air passing across the air-to-liquidheat exchanger (e.g., egressing from the electronics rack into the datacenter).

Referring collectively to FIGS. 11A, 11B & 12, MCU control 1 1210monitors, in one example, temperature of system coolant (T1) egressingfrom MCU 1 1120, as well as flow of facility coolant via a facilitycoolant flow meter (F1) disposed, for example, within MCU 1 1120.Similarly, MCU control 2 1220 monitors temperature of system coolant(T2) egressing from MCU 2 1130, as well as flow of facility coolant (F2)within facility coolant loop 1130 passing through MCU 2. Note that, inan alternative implementation, MCU control 1 and MCU control 2 couldmonitor temperature of facility coolant within the respective facilitycoolant loop 1122, 1132. In the embodiment of FIG. 12, MCU control 1 andMCU control 2 are coupled to a system controller 1200, which itself iscoupled to multiple isolation valves S1, S2 & S3.

As shown in FIG. 11A, isolation valve S1, for example, asolenoid-actuated isolation valve, is disposed in a coolant supply line1161 coupling in fluid communication system coolant supply manifold 1140and air-to-liquid heat exchanger 1160. Isolation valve S2, also by wayof example a solenoid-actuated isolation valve, is disposed (in thisexample) downstream of the system coolant return manifold 1150 so as tobe disposed between the plurality of heat-generating electronicsubsystems 1110 and a coolant return line 1162, which couplesair-to-liquid heat exchanger 1160 to system coolant return manifold1150. Additionally, a system coolant shunt line 1163 is providedcoupling an inlet to air-to-liquid heat exchanger 1160 to between (forexample) system coolant return manifold 1150 and isolation valve S2.Coupled in fluid communication with system coolant shunt line 1163 is athird isolation valve S3, which may also comprise a solenoid-actuatedisolation valve. The system controller 1200 is coupled to isolationvalves S1, S2 & S3 for controlling opening and closing of the isolationvalves, for example, with reference to the monitored temperature T1, T2of system coolant and monitored flow F1, F2 of facility coolant (in oneexample). In the normal-mode of operation depicted in FIG. 11A,isolation valves S1 and S2 are open, and isolation valve S3 is closed.In this arrangement, system coolant flows in parallel to the electronicsubsystems 1110 and the air-to-liquid heat exchanger 1160 for optimalcooling of the electronic components within the electronic subsystemsand the air passing through the electronics rack, as described above.

In the failure-mode of operation depicted in FIG. 11B, isolation valvesS1 and S2 are closed, and isolation valve S3 is open. The systemcontroller automatically transitions the cooling apparatus fromnormal-mode, parallel flow of system coolant through the electronicsubsystems and the air-to-liquid heat exchanger to failure-mode, serialflow of system coolant from the electronic subsystems to theair-to-liquid heat exchanger responsive to, for example, a failure ofthe chilled facility coolant from the source. Specifically, in theexample described above, should chilled facility coolant cooling or flowbe outside design parameters, then monitored system coolant temperatureT1 or T2, and/or monitored facility coolant flow F1 or F2, would resultin a reading indicative of a failure condition, causing the systemcontroller to transition the cooling system from normal-mode tofailure-mode.

FIG. 13 depicts one embodiment of a process for controllingtransitioning of the cooling apparatus of FIGS. 11A, 11B & 12 betweennormal-mode, parallel flow of system coolant through the electronicsubsystems and the air-to-liquid heat exchanger, and failure-mode,serial flow of system coolant from the electronic subsystems to theair-to-liquid heat exchanger, in accordance with an aspect of thepresent invention. In this example, the sensed parameters are systemcoolant temperature T1 and T2, and facility coolant flow rates F1 andF2, associated with MCU 1 and MCU 2, described above and depicted inFIGS. 11A-12. By way of example, T1 and F1 are associated with MCU 1,and T2 and F2 are associated with MCU 2, with each MCU controller beingtied to the higher-level system controller. It is the higher-levelcontroller, based on information received from the MCU controllers, thatwill act to position the isolation valves as warranted (i.e., as oneexample).

The control process of FIG. 13 begins 1300 with the respective MCU 1 andMCU 2 controllers reading T1, F1 & T2, F2 1310, 1320. Before acting onthis data, a check is made to determine the mode of operation of thecooling apparatus 1311, 1321. If other than in the failure-mode (i.e.,the apparatus is in the normal-mode of operation), then processingcompares the sensed parameters against defined set points (orthresholds). Specifically, T1 and T2 are compared against a maximumtemperature threshold (T_(MAX)) 1312, 1322, and facility coolant flowsF1 and F2 (as measured by flow meters F1 & F2, respectively) arecompared against a minimum acceptable facility coolant flow (F_(MIN))1313, 1323. If the temperatures are below T_(MAX) and the flows areabove F_(MIN), then the controller cycles back after waiting a time t1315, 1325 to ascertain a next round of measurements. If either T1 or T2exceeds T_(MAX), or F1 or F2 falls below F_(MIN), then the respectiveMCU 1 or MCU 2 controller signals the system controller to take action1330 with respect to the isolation valves. Responsive to this signal,the system controller closes isolation valve S1 and S2, and opensisolation valve S3 1331, and sets a flag to indicate that the coolingapparatus is now in the failure-mode 1332, after which processing waitstime t 1315, 1325 before obtaining a next round of measurements.

If after measurements are taken 1310, 1320, it is determined that thecooling apparatus is in failure-mode of operation (i.e., “failmode=yes”), then processing determines whether all temperature and flowsatisfy a return set of criteria, that is, a set of criteria forreturning from failure-mode to normal-mode. The set points T_(R) andF_(R) do not necessarily correspond to T_(MAX) and F_(MIN). For example,T_(R) might be lower in magnitude than T_(MAX), and F_(R) could behigher in magnitude than F_(MIN). Note that, unlike the transition tothe failure-mode of operation where any one parameter could trigger theswitch, in order to return back to normal-mode of operation, allparameters must meet the test criteria. Therefore, in failure-mode,processing initially determines whether T1<T_(R), and F1>F_(R) 1340, anddetermines whether T2<T_(R), and F2>F_(R) 1350. If inquiry 1340 is “no”,processing waits time 1315 before returning to obtain a next set ofmeasurements, while if inquiry 1350 is “no”, then processing waits atime t 1380 before returning to collect a next set of measurements 1320.Note that inquiry 1340 might result in a “yes”, and inquiry 1350 mightresult in a “no” or vice versa. In order to proceed, both inquiries mustbe “yes” 1360. Thus, for a “yes” inquiry 1340, 1350, where the otherinquiry is “no”, processing returns from 1360 to wait time t 1315 orwait time t 1380, depending upon whether the “yes” from the inquiry wasfor MCU 1 or MCU 2. Assuming that all tests are met for both MCUs 1360,then the system controller is notified to take action 1365 to openisolation valves S1 and S2, and close isolation valve S3 1370, afterwhich the failure-mode flag is set back to “no”, indicating normal-modeof operation 1375, and the process loop continues.

Those skilled in the art should note from the above description that thecontrol process of FIG. 13, as well as the cooling apparatus of FIGS.11A-12, can be readily modified to employ a single MCU, oralternatively, more than two MCUs, if desired.

As noted above, as component density within servers continues toincrease to achieve increased performance, heat generated withinelectronic systems necessitates liquid cooling in some cases. Energyefficiency is a significant feature for all system designs, whether airor liquid-cooled, and increased coolant flow rates typically increasesthe energy use of the cooling system. The increased reliability andperformance gains associated with cooler running of electroniccomponents (such as CMOS processors), is known, and techniques such asvapor compression refrigeration have been employed at the cost of energyefficiency. Thus, a solution offering energy efficient cooling at (forexample) a sub-ambient temperature, ultimately sinking to air or aliquid coolant with built-in redundancy, is believed desirable. Oneembodiment of such a cooling apparatus is described below, by way ofexample only, with respect to FIGS. 14-16.

Generally stated, disclosed herein in another aspect, is a coolingapparatus which includes: one or more liquid-cooled structuresassociated with one or more electronic components to be cooled; acoolant loop which includes a first loop portion and a second loopportion coupled in parallel, wherein the one or more liquid-cooledstructures are coupled in fluid communication with the first loopportion of the coolant loop; a liquid-to-liquid heat exchanger and anair-to-liquid heat exchanger coupled in series fluid communication bythe coolant loop, with coolant egressing from the liquid-to-liquid heatexchanger passing via the coolant loop through the air-to-liquid heatexchanger; and a thermoelectric array including one or morethermoelectric modules. The thermoelectric array is disposed with thefirst loop portion of the coolant loop at least partially in thermalcontact with a first side of the thermoelectric array, and the secondloop portion of the coolant loop at least partially in thermal contactwith a second side of the thermoelectric array. The thermoelectric arrayoperates to transfer heat from coolant passing through the first loopportion to coolant passing through the second loop portion, wherein thethermoelectric array cools coolant passing through the first loopportion before the coolant passes through the one or more liquid-cooledstructures, and after passing through the one or more liquid-cooledstructures, the coolant passing through the first loop portion and thecoolant passing through the second loop portion pass through theseries-coupled, liquid-to-liquid heat exchanger and air-to-liquid heatexchanger. In operation, the liquid-to-liquid heat exchanger or theair-to-liquid heat exchanger operates as heat sink for the coolant loop,depending on a mode of operation of the cooling apparatus. A controlleris coupled to the thermoelectric array and automatically adjustsoperation of the thermoelectric array between a liquid-cooled mode andan air-cooled mode depending, at least in part, on an operational stateof the liquid-to-liquid heat exchanger. When in an air-cooled mode, thecontroller operates the thermoelectric array to transfer greater heat(from the coolant passing through the first loop portion to the coolantpassing through the second loop portion) than in the liquid-cooled mode.In the liquid-cooled mode, the liquid-to-liquid heat exchanger operatesas the heat sink for the coolant loop, and in the air-cooled mode, theair-to-liquid heat exchanger operates as heat sink for the coolant loop.

Advantageously, disclosed below in connection with FIGS. 14-16 is acooled electronic system and a cooling apparatus which employ athermoelectric array (also referred to herein as athermoelectric-enhanced, fluid-to-fluid heat exchange assembly). One ormore coolant pumps pressurize coolant, such as water, causing it to flowthrough the first and second loop portions of the coolant loop inthermal contact with the first and second sides of thethermoelectric-enhanced, fluid-to-fluid heat exchange assembly. Aftercooling the coolant in the first loop portion, the coolant subsequentlyflows through one or more liquid-cooled structures, such as one or moreliquid-cooled cold plates, coupled to one or more electronic componentsto be cooled. Downstream of the electronic component(s), the coolantexhaust meets the coolant exhaust that was in thermal contact with thehot side of the thermoelectric-enhanced, fluid-to-fluid heat exchangeassembly, that is, the coolant that passed through the second loopportion. In one embodiment, a flow rate through the second loop portionis metered by a proportional valve or a second loop control valve. Thejoined coolant then flows through the liquid-to-liquid heat exchanger,which in one embodiment is cooled via a facility chilled water source ina manner similar to that described above. After passing through theliquid-to-liquid heat exchanger, the coolant flows through anair-to-liquid heat exchanger, which in one embodiment, is disposed atthe air outlet side of an associated electronics rack to cool airflowexhausting from the electronics rack. The system coolant is thenreturned to the pump for continued recirculation through the coolantloop.

In the case where facility coolant (e.g., building-chilled water) isunavailable (for example, the system is intended to be run in theair-cooled mode) or the facility coolant has failed (as describedabove), coolant exhausting from the liquid-cooled structure and thesecond loop portion passes through the liquid-to-liquid heat exchangerto the air-to-liquid heat exchanger, where the airflow across theair-to-liquid heat exchanger (e.g., the airflow egressing from theassociated electronics rack) cools the coolant within the coolant loop.Note that in this air-cooled mode, the thermoelectric array requiresgreater electrical current (from, for example, an adjustable powersupply (not shown)) to pump more heat from the first, cold side of thearray to the second, hot side of the array (i.e., from coolant withinthe first loop portion to the coolant within the second loop portion).This negatively affects energy efficiency, but still supplies coolant ata desired or appropriate temperature to the one or more liquid-cooledstructures.

FIG. 14 depicts one embodiment of a cooled electronic system, generallydenoted 1400, in accordance with one or more aspects of the presentinvention. The cooled electronic system includes, by way of example, anelectronics rack 1401 with one or more electronic systems or subsystems1402 comprising air-cooled components, and one or more air-movingdevices 1403, such as one or more fans, for moving air from an air inletside 1404 to an air outlet side 1405 of electronics rack 1401 to coolthe air-cooled components thereof. After cooling the air-cooledcomponents of the electronic system(s) 1402, and the airflow egresses asheated airflow 1406.

A cooling apparatus is provided which includes one or more liquid-cooledstructures 1410 and a coolant loop 1420. The one or more liquid-cooledstructures 1410 are coupled to one or more electronic components to becooled, such as high-heat-generating electronic components of theelectronic system(s) 1402. The coolant loop 1420 includes a first loopportion 1421 and a second loop portion 1422, which are disposed asparallel portions of the coolant loop. The liquid-cooled structure iscoupled in fluid communication with or within the first loop portion1421 of the coolant loop 1420. The cooling apparatus further includes,by way of example, a liquid-to-liquid heat exchanger 1430 and anair-to-liquid heat exchanger 1440. The liquid-to-liquid heat exchanger1430 and air-to-liquid heat exchanger 1440 are coupled in series fluidcommunication via coolant loop 1420, as illustrated in FIG. 14. In thisconfiguration, coolant egressing from liquid-to-liquid heat exchanger1430 passes, via coolant loop 1420, to air-to-liquid heat exchanger1440.

In one embodiment, liquid-to-liquid heat exchanger 1430 is configured totransfer heat from system coolant passing through coolant loop 1420 tofacility coolant passing through a facility coolant loop 1431. Afacility coolant control valve PV1 (e.g., a proportional valve) isprovided in fluid communication with facility coolant loop 1431 tocontrol flow of facility coolant through liquid-to-liquid heat exchanger1430 via (in one embodiment) a control process implemented by acontroller 1480. In implementation, controller 1480 may be associatedwith electronics rack 1401, or alternatively, may be centrally disposedwithin a data center monitoring multiple cooling apparatuses associatedwith multiple electronics racks of the data center.

As noted, depending on the mode of operation, facility coolant may ormay not be flowing through liquid-to-liquid heat exchanger 1430. Forinstance, in a data center installation lacking facility coolant, thenthe cooling apparatus is operating in air-cooled mode and the systemcoolant simply passes through liquid-to-liquid heat exchanger 1430, withthe heat sink being air-to-liquid heat exchanger 1440. Alternatively, iffacility coolant is available and the facility coolant system isoperating, then the cooling apparatus is operated in liquid-cooled modesince the facility coolant will be at a lower temperature than theheated airflow 1406 exhausting from electronics rack 1401. Note that, asan alternate embodiment, the air-to-liquid heat exchanger 1440 may bedisposed at other than the air outlet side of electronics rack 1401. Forexample, the heat exchanger could be disposed in a parallel airflow pathto the airflow through the electronics rack, if desired.

A thermoelectric array or thermoelectric-enhanced, fluid-to-fluid heatexchange assembly 1450 is also provided as part of the coolingapparatus, with a first, cold side of the thermoelectric array coupledin thermal contact with first loop portion 1421 of coolant loop 1420,and the second, hot side of the thermoelectric array coupled in thermalcontact with the second loop portion 1422 of coolant loop 1420. Inparticular, and in one embodiment, the thermoelectric array may includea first heat exchange element 1451 and a second heat exchange element1452 on opposite sides of the thermoelectric array, which facilitatetransfer of heat from coolant passing through the first loop portion,across the thermoelectric modules of the thermoelectric array, tocoolant within the second loop portion 1422 of coolant loop 1420.Configuration and operation of thermoelectric array 1450 are discussedfurther below in connection with FIG. 15.

Continuing with FIG. 14, a second loop control valve PV2 (for example, asecond proportional valve) is associated with second loop portion 1422for controlling coolant flow through second loop portion 1422, asdescribed below. One or more coolant pumps 1470 are in fluidcommunication with coolant loop 1420, and provide for flow of coolantthrough the coolant loop. Multiple temperature sensors may be provided,including: a first temperature sensor T1 sensing coolant inlettemperature to the one or more coolant-cooled structures 1410; a secondtemperature sensor T2 sensing temperature of coolant to theair-to-liquid heat exchanger 1440; a third temperature sensor T3 sensingtemperature associated with the thermoelectric array 1450, such as atthe second, hot side thereof; and a fourth temperature sensor T4associated with one or more of the air-cooled components of theelectronic system(s) 1402 being air-cooled (alternatively, sensor T4could sense air temperature across the air-cooled components). Thesetemperature sensors facilitate automated control of the cooling mode,coolant temperature, coolant flow rate, etc., through the coolingapparatus depicted in FIG. 14, for example, in a manner such asdescribed below in connection with the control process of FIG. 16. Inaddition to the facility loop control valve PV1 and second loop controlvalve PV2, a differential pressure sensor (dP) may be providedassociated with first loop portion 1421 for measuring (for instance)differential pressure across the first heat exchange element 1451 of thethermoelectric array 1450. This differential pressure may be used, forexample, in an automated process for controlling operational speed ofthe one or more pumps 1470.

Note that the particular sensors and control points, as well as theirlocations, are provided herein by way of example only, and that multipleadditional sensors or control points may be employed without departingfrom the scope of the present invention.

As noted, FIG. 15 is a cross-sectional elevational view of oneembodiment of a thermoelectric-enhanced, fluid-to-fluid heat exchangeassembly 1450, in accordance with an aspect of the present invention. Inthis example, first heat exchange element 1451 is, as one example, aliquid-cooled cold plate, and second heat exchange element 1452 is asecond liquid-cooled cold plate, wherein coolant through first andsecond loop portions of the coolant loop ingresses and egresses throughthe respective cold plate. Thermoelectric array 1450 comprises, in thisexample, an array 1500 of thermoelectric modules 1501, each of whichcomprises individual thermoelectric elements 1502.

The use of large thermoelectric cooling elements is known. Theseelements operate electronically to produce a cooling effect. By passinga direct current through the legs of a thermoelectric device, a heatflow is produced across the device which may be contrary to that whichwould be expected from Fourier's law.

At one junction of the thermoelectric element, both holes and electronsmove away, towards the other junction, as a consequence of the currentflow through the junction. Holes move through the p-type material andelectrons through the n-type material. To compensate for this loss ofcharge carriers, additional electrons are raised from the valence bandto the conduction band to create new pairs of electrons and holes. Sinceenergy is required to do this, heat is absorbed at this junction.Conversely, as an electron drops into a hole at the other junction, itssurplus energy is released in the form of heat. This transfer of thermalenergy from the cold junction to the hot junction is known as thePeltier effect.

Use of the Peltier effect permits the surfaces attached to a heat sourceto be maintained at a temperature below that of a surface attached to aheat sink. What these thermoelectric modules provide is the ability tooperate the cold side below the ambient temperature of the coolingmedium (e.g., air or water). When direct current is passed through thethermoelectric modules, a temperature difference is produced with theresult that one side is relatively cooler than the other side. Thesethermoelectric modules are therefore seen to possess a hot side and acold side, and provide a mechanism for facilitating the transfer ofthermal energy from the cold side of the thermoelectric module to thehot side of the thermoelectric module.

By way of specific example, thermoelectric modules 1501 may comprise TECCP-2-127-06L modules, offered by Melcor Laird, of Cleveland, Ohio.

Note that the thermoelectric array may comprise any number ofthermoelectric modules, including one or more modules, and is dependent(in part) on the size of the electronic modules, as well as the amountof heat to be transferred from coolant flowing through first heatexchange element 1451, to coolant flowing through second heat exchangeelement 1452. Also note that an insulative material (not shown) may beprovided over one or more of the exposed surfaces of first heat exchangeelement 1451 or second heat exchange element 1452.

The thermoelectric (TE) array may comprise a planar thermoelectric arraywith modules arranged in a square or rectangular array. Although thewiring is not shown, each thermoelectric module in a column may be wiredand supplied electric current (I) in series and the columns ofthermoelectric modules may be electrically wired in parallel so that thetotal current supplied would be I×sqrt(M) for a square array comprisingM thermoelectric modules, providing an appreciation of the inherentscalability of the array. In this way, if a single thermoelectric moduleshould fail, only one column is effected, and electric current to theremaining columns may be increased to compensate for the failure.

Table 1 provides an example of the scalability provided by a planarthermoelectric heat exchanger configuration such as described herein.

TABLE 1 Number of TE Modules (M) Heat Exchanger Size 81 585 mm × 585 mm(23.0 in. × 23.0 in.) 100 650 mm × 650 mm (25.6 in. × 25.6 in.) 121 715mm × 715 mm (28.2 in. × 28.2 in.) 144 780 mm × 780 mm (30.7 in. × 30.7in.) 169 845 mm × 845 mm (33.3 in. × 33.3 in.)

For a fixed electric current and temperature difference across thethermoelectric modules, the heat pumped by the thermoelectric array willscale with the number of thermoelectric modules in the platform area.Thus, the heat load capability of a 650 mm×650 mm thermoelectric heatexchanger will be 1.23 times that of a 585 mm×585 mm thermoelectric heatexchanger, and that of an 845 mm×845 mm will be 2.09 times greater. Notethat the size of the liquid-to-air heat exchanger may need to grow toaccommodate the increased heat load. If the space available for thethermoelectric heat exchanger is constrained in the X×Y dimensions, thenthe heat pumping capabilities can still be scaled upwards by growing inthe Z dimension. This can be done by utilizing multiple layers ofthermoelectric modules between multiple heat exchange elements, withalternating hot and cold sides, as described, for example, in U.S. Pat.No. 6,557,354 B1.

Continuing with FIG. 14, in operation, the cooling apparatus disclosedcools, at least in part, one or more electronic components via the oneor more liquid-cooled structures. Coolant passing through the one ormore liquid-cooled structures is cooled to a desired temperature via thesolid state thermoelectric array 1450, which comprises one or morethermoelectric modules with a first, cold side where heat is removedfrom the coolant, and a second, hot side where heat is transported tocoolant in separate flow paths of the coolant loop.

In a liquid-cooled mode of operation, the heat is transferred to asecond coolant (referred to herein as the facility coolant) flowingthrough one side of the liquid-to-liquid heat exchanger. The secondcoolant has a flow rate metered by, for example, a proportional valve ora diverter valve, which controls flow between the liquid-to-liquid heatexchanger and a parallel bypass flow path (not shown) to maintain thetemperature of the system coolant within the coolant loop leaving theliquid-to-liquid heat exchanger within a desired range. In thisliquid-cooled mode, the cooled coolant flowing from the liquid-to-liquidheat exchanger to the air-to-liquid heat exchanger cools (in oneembodiment) the airflow 1406 egressing from the electronics rack. Thecoolant exiting the air-to-liquid heat exchanger enters the coolantcirculation pump (or pumps). Downstream of the coolant pump(s), thecoolant flow is split into the two parallel coolant streams, that is, toflow through the first loop portion and the second loop portion. Asnoted, coolant flowing through the first loop portion is in thermalcontact with the first, cold side of the thermoelectric array, and iscooled to a specified temperature by varying the electrical current flowthrough the thermoelectric modules of the array responsive to, forexample, the coolant temperature T1 supplied to the one or moreliquid-cooled structures. The second stream flows via the second loopportion, and is in thermal contact with the second, hot side of thethermoelectric array, dissipating heat removed from the cold side, aswell as any losses within the thermoelectric modules of the array. Thecoolant cools the one or more electronic components via theliquid-cooled structure(s) conductively coupled to the electroniccomponent(s), and subsequently rejoins the coolant flowing through thesecond loop portion of the coolant loop. Coolant flow through the secondloop portion is controlled by, for example, a proportional valve orother flow control valve PV2 responsive to, for example, temperature T3of the thermoelectric modules to maintain their operating temperaturebelow a specified threshold for reliable operation. The combined coolantflow then progresses back to the liquid-to-liquid heat exchanger totransfer the system's heat load to the second coolant (i.e., facilitycoolant).

In the air-cooled mode, which in accordance with an aspect of thepresent invention may be either a failure mode or an air-cooled onlymode (e.g., where facility coolant is unavailable in the data center),the liquid-to-liquid heat exchanger does not affect the coolant flow, ormore particular, there is little or no heat transferred from thencoolant within the liquid-to-liquid heat exchanger, since there is nofacility coolant flowing through the facility coolant loop or facilitycoolant side of the liquid-to-liquid heat exchanger. In such a case, thecombined returning coolant from the thermoelectric array and the secondloop portion of the coolant loop will be at a higher temperature thanthe airflow egressing from the electronics rack. In this mode, heat willthus be transferred from the liquid coolant to the exhaust air, and theelectrical current flow to the thermoelectric array will need to behigher than in the liquid-cooled mode in order to obtain the desiredcoolant temperature T1 to cool the one or more electronic components viathe liquid-cooled structure(s). The resulting temperature at the second,hot side of the thermoelectric array will be higher as well. Also, inthe air-cooled mode, the thermoelectric array will consume more power,thereby reducing energy efficiency of the cooling apparatus, but theliquid-cooled structure(s), and thus the electronic component(s) coupledthereto, will be cooled to the same extent as in the liquid-cooled mode.

FIG. 16 depicts one example of a control process implemented, forexample, by a controller such as controller 1480 of the coolingapparatus of FIG. 14. Upon initiation of the control process 1600,temperatures are read via temperature sensors T1, T2, T3 & T4, anddifferential pressure across the first heat exchange element of thethermoelectric array is read, along with positioning of the facilitycoolant control valve (PV1), the second loop control valve (PV2), andoperating speed (RPM) of the one or more coolant pumps facilitating flowof coolant through the coolant loop 1605. With this information, thecontroller inquires (e.g., in parallel) whether temperature T1 is withina specified operational range 1610 (for instance, between a specifiedhigh temperature (T1 _(SPEC,HIGH)) and a specified low temperature (T1_(SPEC,LOW))), whether temperature T4 is within specification (forinstance, between a specified high temperature (T4 _(SPEC,HIGH)) and aspecified low temperature (T4 _(SPEC,LOW))) 1640, and whethertemperature T3 is within specification (for instance, between aspecified high temperature (T3 _(SPEC,HIGH)) and a specified lowtemperature (T3 _(SPEC,LOW))) 1650.

Beginning with temperature T1, if temperature T1 is withinspecification, no action is taken, and processing returns to againobtain the above-noted sensor readings 1605. If desired, a wait timeinterval may be interposed within this return loop. If coolanttemperature T1 into the liquid-cooled structure(s) is outside ofspecification, then processing determines whether the facility coolantcontrol valve PV 1 is between its specified maximum (PV1 _(MAX)) andminimum (PV1 _(MIN)) values 1615. If so, then the controller adjustsvalve PV1 responsive to the coolant temperature T1 1620 to eitherincrease or decrease the flow of facility coolant through theliquid-to-liquid heat exchanger. If the facility coolant control valvePV 1 is either at maximum (PV1 _(MAX)) or minimum (PV1 _(MIN)) setting,then the controller automatically adjusts the voltage (V_(TE)) providedto the thermoelectric array 1625, and determines whether coolanttemperature T2 is greater than air temperature T4 1630. If so, then theone or more air-moving devices associated with the electronics rack areincreased to a “high” setting 1635. In one embodiment, this “high”setting is less than a maximum setting, and is, for instance, aspecified high speed setting which allows continued operation of the oneor more air-moving devices. If coolant temperature T2 is equal or lessthan air temperature T4, then processing returns to obtain the next setof readings 1605.

If the airflow temperature T4 across the air-cooled components is withinspecification, then from inquiry 1640 no action is taken, and processingobtains a next set of readings 1605. However, if the airflow temperatureis outside of specification, that is, is either at or above thespecified high temperature (T4 _(SPEC,HIGH)), or is at or below thespecified low temperature (T4 _(SPEC,LOW)), then the controllerautomatically adjusts the operating speed of the one or more air-movingdevices, responsive to the sensed temperature T4 1645. For instance, iftemperature T4 is too high, then air-moving device rotational speed isincreased, or if temperature T4 is too low, then the speed is reduced.

If the temperature T3 of the thermoelectric array is withinspecification 1650, then processing adjusts the second loop controlvalve PV2 to adjust flow of coolant through the second loop portion inthermal contact with the thermoelectric array responsive to temperatureT3 1655 to, for example, maintain the temperature near a desired setpoint within the range between the specified high temperature (T3_(SPEC,HIGH)) and specified low temperature (T3 _(SPEC,LOW)). Afteradjusting the coolant flow rate through the second loop portion, thecontroller adjusts the speed of the coolant pump responsive to thepressure differential across the first heat exchange element of thethermoelectric-enhanced, fluid-to-fluid heat exchange assembly 1660.

If the thermoelectric temperature T3 is outside of specification, thenthe controller determines whether the second loop control valve PV2 isless than maximum 1665. If “no”, then coolant pump operating speed isincreased to a maximum 1670 to attempt to further reduce temperature T3.If flow control valve PV2 is set less than maximum open, then processingdetermines whether the operating speed of the coolant pump(s) is lessthan a maximum speed (RPM_(PUMP,MAX)) 1675. If the coolant pump(s) areoperating at maximum, then the second loop control valve PV2 is openedto maximum 1680. Otherwise, the flow control valve PV2 is adjustedresponsive to the temperature T3 of the thermoelectric array 1655, afterwhich the control adjusts pump(s) speed responsive to the differentialpressure (dP) across the first heat exchange element of thethermoelectric-enhanced, fluid-to-fluid heat exchange assembly.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system”.Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readable signalmedium may include a propagated data signal with computer readableprogram code embodied therein, for example, in baseband or as part of acarrier wave. Such a propagated signal may take any of a variety offorms, including, but not limited to, electro-magnetic, optical or anysuitable combination thereof. A computer readable signal medium may beany computer readable medium that is not a computer readable storagemedium and that can communicate, propagate, or transport a program foruse by or in connection with an instruction execution system, apparatusor device.

A computer readable storage medium may be, for example, but not limitedto, an electronic, magnetic, optical, electromagnetic, infrared orsemiconductor system, apparatus, or device, or any suitable combinationof the foregoing. More specific examples (a non-exhaustive list) of thecomputer readable storage medium include the following: an electricalconnection having one or more wires, a portable computer diskette, ahard disk, a random access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM or Flash memory), anoptical fiber, a portable compact disc read-only memory (CD-ROM), anoptical storage device, a magnetic storage device, or any suitablecombination of the foregoing. In the context of this document, acomputer readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Referring now to FIG. 17, in one example, a computer program product1700 includes, for instance, one or more computer readable storage media1702 to store computer readable program code means or logic 1704 thereonto provide and facilitate one or more aspects of the present invention.

Program code embodied on a computer readable medium may be transmittedusing an appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programminglanguage, such as Java, Smalltalk, C++ or the like, and conventionalprocedural programming languages, such as the “C” programming language,assembler or similar programming languages. The program code may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider).

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

In addition to the above, one or more aspects of the present inventionmay be provided, offered, deployed, managed, serviced, etc. by a serviceprovider who offers management of customer environments. For instance,the service provider can create, maintain, support, etc. computer codeand/or a computer infrastructure that performs one or more aspects ofthe present invention for one or more customers. In return, the serviceprovider may receive payment from the customer under a subscriptionand/or fee agreement, as examples. Additionally or alternatively, theservice provider may receive payment from the sale of advertisingcontent to one or more third parties.

In one aspect of the present invention, an application may be deployedfor performing one or more aspects of the present invention. As oneexample, the deploying of an application comprises providing computerinfrastructure operable to perform one or more aspects of the presentinvention.

As a further aspect of the present invention, a computing infrastructuremay be deployed comprising integrating computer readable code into acomputing system, in which the code in combination with the computingsystem is capable of performing one or more aspects of the presentinvention.

As yet a further aspect of the present invention, a process forintegrating computing infrastructure comprising integrating computerreadable code into a computer system may be provided. The computersystem comprises a computer readable medium, in which the computermedium comprises one or more aspects of the present invention. The codein combination with the computer system is capable of performing one ormore aspects of the present invention.

Although various embodiments are described above, these are onlyexamples. For example, computing environments of other architectures canincorporate and use one or more aspects of the present invention.Additionally, the network of nodes can include additional nodes, and thenodes can be the same or different from those described herein. Also,many types of communications interfaces may be used. Further, othertypes of programs and/or other optimization programs may benefit fromone or more aspects of the present invention, and other resourceassignment tasks may be represented. Resource assignment tasks includethe assignment of physical resources. Moreover, although in one example,the partitioning minimizes communication costs and convergence time, inother embodiments, the cost and/or convergence time may be otherwisereduced, lessened, or decreased.

Further, a data processing system suitable for storing and/or executingprogram code is usable that includes at least one processor coupleddirectly or indirectly to memory elements through a system bus. Thememory elements include, for instance, local memory employed duringactual execution of the program code, bulk storage, and cache memorywhich provide temporary storage of at least some program code in orderto reduce the number of times code must be retrieved from bulk storageduring execution.

Input/Output or I/O devices (including, but not limited to, keyboards,displays, pointing devices, DASD, tape, CDs, DVDs, thumb drives andother memory media, etc.) can be coupled to the system either directlyor through intervening I/O controllers. Network adapters may also becoupled to the system to enable the data processing system to becomecoupled to other data processing systems or remote printers or storagedevices through intervening private or public networks. Modems, cablemodems, and Ethernet cards are just a few of the available types ofnetwork adapters.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise” (andany form of comprise, such as “comprises” and “comprising”), “have” (andany form of have, such as “has” and “having”), “include” (and any formof include, such as “includes” and “including”), and “contain” (and anyform contain, such as “contains” and “containing”) are open-endedlinking verbs. As a result, a method or device that “comprises”, “has”,“includes” or “contains” one or more steps or elements possesses thoseone or more steps or elements, but is not limited to possessing onlythose one or more steps or elements. Likewise, a step of a method or anelement of a device that “comprises”, “has”, “includes” or “contains”one or more features possesses those one or more features, but is notlimited to possessing only those one or more features. Furthermore, adevice or structure that is configured in a certain way is configured inat least that way, but may also be configured in ways that are notlisted.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiment was chosen and described in order to best explain theprinciples of one or more aspects of the invention and the practicalapplication, and to enable others of ordinary skill in the art tounderstand one or more aspects of the invention for various embodimentswith various modifications as are suited to the particular usecontemplated.

What is claimed is:
 1. A method of fabricating a cooling apparatuscomprising: providing a liquid-cooled structure, the liquid-cooledstructure being configured to couple to at least one electroniccomponent to be cooled; providing a coolant loop, the coolant loopcomprising a first loop portion and a second loop portion, the firstloop portion and the second loop portion being parallel portions of thecoolant loop; coupling the liquid-cooled structure in fluidcommunication with the first loop portion of the coolant loop; providinga liquid-to-liquid heat exchanger and an air-to-liquid heat exchangercoupled in series fluid communication via the coolant loop, whereincoolant egressing from the liquid-to-liquid heat exchanger passes, viathe coolant loop, through the air-to-liquid heat exchanger; andproviding a thermoelectric array comprising at least one thermoelectricmodule, the thermoelectric array being disposed with the first loopportion of the coolant loop at least partially in thermal contact with afirst side of the thermoelectric array, and the second loop portion ofthe coolant loop at least partially in thermally contact with a secondside of the thermoelectric array, wherein the thermoelectric arrayoperates to transfer heat from coolant passing through the first loopportion to coolant passing through the second loop portion, thethermoelectric array cooling coolant passing through the first loopportion before the coolant passes through the liquid-cooled structure,and after passing through the liquid-cooled structure, the coolantpassing through the first loop portion and the coolant passing throughthe second loop portion pass through the series-coupled,liquid-to-liquid heat exchanger and air-to-liquid heat exchanger,wherein one of the liquid-to-liquid heat exchanger or the air-to-liquidheat exchanger operates as heat sink for the coolant loop, dependent ona mode of operation of the cooling apparatus.
 2. The method of claim 1,further comprising providing a controller coupled to the thermoelectricarray and configured to automatically adjust operation of thethermoelectric array between a liquid-cooled mode and an air-cooledmode, depending, at least in part, on an operational state of theliquid-to-liquid heat exchanger, wherein in the air-cooled mode, thecontroller operates the thermoelectric array to transfer greater heatfrom coolant passing through the first loop portion to coolant passingthrough the second loop portion than in the liquid-cooled mode, andwherein in the liquid-cooled mode, the liquid-to-liquid heat exchangeroperates as heat sink for the coolant loop, and in the air-cooled mode,the air-to-liquid heat exchanger operates as the heat sink for thecoolant loop.
 3. The method of claim 2, further comprising: providing afacility loop control valve coupled to a facility control loop to supplyfacility coolant to the liquid-to-liquid heat exchanger, wherein thecontroller automatically adjusts the facility loop control valveresponsive to a temperature of coolant supplied to the coolant-cooledstructure, via the first loop portion, being outside of a specifiedoperational range, and responsive to the facility loop control valvebeing outside of a specified control range, the controller automaticallyadjusts voltage supplied to the thermoelectric array to move thetemperature of the coolant supplied to the coolant-cooled structure backtowards its specified operational range; and providing at least oneair-moving device for facilitating airflow across the air-to-liquid heatexchanger, wherein the controller automatically adjusts rotational speedof the at least one air-moving device responsive to the temperature ofthe coolant supplied to the air-to-liquid heat exchanger being greaterthan the temperature of the airflow passing across the air-to-liquidheat exchanger.
 4. The method of claim 2, further comprising providingat least one air-moving device facilitating airflow across theair-to-liquid heat exchanger, wherein the controller automaticallyadjusts rotational speed of the at least one air-moving deviceresponsive to temperature of the airflow across the air-to-liquid heatexchanger being outside of a specified air temperature range.
 5. Themethod of claim 2, further comprising: providing a second loop controlvalve coupled in fluid communication with the second loop portion of thecoolant loop and controlled by the controller, wherein the controllerautomatically adjusts flow of coolant through the second loop portion ofthe coolant loop via the second loop control valve dependent on a sensedtemperature at the thermoelectric array; providing at least one coolantpump coupled in fluid communication with the coolant loop for, at leastin part, pumping coolant in parallel through the first loop portion andthe second loop portion of the coolant loop, wherein the controllerautomatically adjusts operation of the at least one pump responsive to achange in coolant pressure ascertained across at least a portion of thefirst loop portion due to an adjustment of coolant flow through thesecond loop portion of the control loop; and wherein responsive to thesensed temperature at the thermoelectric array being outside of aspecified temperature range, the controller automatically adjustsoperating speed of the at least one coolant pump to a maximum, andresponsive to the at least one coolant pump already being at a maximumoperating speed, the controller automatically adjusts the second loopcontrol valve to maximum open.